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ABSTRACT 

Issues in the ongoing evaluation of an innovative 
curriculum in an academic setting are discussed. The program provides 
instruction in computer literacy to a physically challenged clientele 
and to professionals from local service agencies who have 
disabilities. Data have t^en collected through survey questionnaires 
principally composed of Likert items and direct or telephone 
interviews. Both interviews deal with: (1) likes and disliXes of 
participants? (2) desired changes, (3) perception of effects on 
individual improvement? and (4) demographics/client background data. 
The foremost problem has been that of unintentional alienation of 
subjects due to semantics in talking about disabilities- Face-to-face 
interviews with visually impaired subjects brought different problems 
in terms of potential bias. Confidentiality is a problem transcending 
all types of evaluation instrumentation » Another important issue i.gis 
t^en the utilization of the evaluation's findings. The evaluation of 
the first year was summative? it served as the basis for fomative 
changes for subsequent years, strategies for the successful conduct 
of this summative evaluation with formative purposes must take the 
acknowledged problems into account. Evaluating a program for special 
needs groups may require special preparation and sensitivity, 
(SLD) 
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The opinioris expressed in this paper are the results of the 
give and take between the two authors based on their mutual 
experience and research. The order of authors' names is 
alphabetical. No assumption of first authorship should be made. 



EVALUATION INSTRUMENTS AND PHYSICALLY CHALLENGED SUBJECTS: 

PROBLEMS, ISSUES AND STRATEGIES 



INTRODUCTIUN — 

In order for program evaluation to effectively render a 
judgment of merit or worth (Madaus, Scriven and stuf f lebeam, 
1988), it is important that strategic planning is focused on the 
situation being evaluated. Unique challenges are encountered in 
working with those programs which purport to serve the special 
needs of physically challenged populations. Evaluators face a 
continual learning process in striving to meet whatever standards 
of effective evaluation they are aiming for. The authors believe 
that relevant experiences should be shared in the evaluation 
community to render a more realistic and responsive range of 
measurement techniques. 

The basis for this paper will be the ongoing evaluation of 
an innovative curriculum program in an academic setting. This 
program provides instruction in computer literacy to both a 
physically challenged clientele and disability professionals from 
local service agencies. It is the measurement of the affective 
dati? sought from the challenged population that has provided thct 
impetus for this line of reasoning. 
FOCUS 

Throughout the evaluation, data has been collected from 
program participants using both survey questionnaires (in a 
pretest-posttest non-randomized design) as well as direct or 
telephone interviews. The survey instrument has been piloted 
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bo fore each annual use. The evaluation is now in its second 
year; it ideally has a different set of respondents or a new 
class section in every semester. Although validity may tend to 
be subjective (House, 1978) when dealing with affective data, it 
continues to be a concern of the evaluation team. 

Survey questions consist principally of Likert items using a 
seven point semantic differential scale for various key issues. 
There are also a number of open ended opinion questions, many of 
which wore added for the second (this current) year of the 
project based on perceived needs uncovered during year one. 
Interview questions can be open ended or may seek the classic 
yes/ no answer. At any rate, they have sought to employ a 
conversational tone based on the evaluators' cognizance of the 
value of multiple perspectives (Madaus, Scriven, Stuf f lebeam, 
1988) . LiKc most evaluations, there are a core group of 
anticipated or desired outcomes, especially on the part of the 
sponsor. The evaluation team has done its best to avoid gettimj 
caught up in this phenomenon. It is, however, tied in somewhat 
unavoidably to the content of the instruments in question. This 
is primarily due to the fact that program standards are based on 
a grant proposal written by the sponsor well before this 
evaluation. The program's funding is by and large external. 

This emphasis is implicit in roost of the sux'vey and 
interview questions as they deal with: a) the main likes and 
dislikes of participants about the program? b) desired changes,* 
and, c) perception of course effect on individual improvement (s) . 
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Aside from these broad conceptual areas, the bulk of remaining 
questions asked deal with demographics or client background data. 
PROBLEMS / ANALYSIS 

In administering the written survey instrument to the 
designated sample of physically challenged subjects, the foremoat 
problem was that of unintentional alienation due to semantics. 
This alienation can be subtle , yet it can pose a barrier to tno 
essential level of rapport between evaluator and respondent. For 
example, one question asked for a description of the applicable 
"disability." In retrospect, we see a low response rate and 
little utility to the question. And, of those who did respond, 
some simply indicated that they are not "disabled." 

Clearly the problem here is with the use of terminology. 
The very word "disabled" may border on being offensive to some of 
the clients. To others it simply does not apply; the feeling is 
that they are indeed "able" to attack their circumstances by a 
strategy such as this course (Horne, 1988). it is not only 
important to establish a viable common language (Converse and 
Pressor, 1986) but also to make that language acceptable and non- 
alienating. 

Comprehensibility of certain items can also be problematic. 
In developing Likert scale items, it is necessary to give 
specific and understandable values to each point as well .is a 
concise and explicit set of instructions at the start. Even it 
this may seem redundant to the evaluator, it must be remembered 
that the physically challenged person may have a constant 
struggle with such dilemmas. This may be especially true for 
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people with a reading or perceptual problem. If the preservation 
of conceptual clarity is among the investigator's most difficult 
challenges (Converse and Presser, 1986) , then the degree of 
difficulty is multiplied when trying to accommodate special 
needs. Similarly, length of the questions and the overall 
questionnaire can adversely effect the data yield by discouraging 
completion. Though the importance of brevity in this type of 
activity was pointed out by Payne some time ago (Converse, 
Presser, 1989) , this issue is accentuated for those who have 
difficulty in writing. 

Face to face interviewing using the same survey questions at*, 
those on the written questionnaire was necessitated in the cases 
of visually impaired subjects. A different set of problems aroFc 
here in terms of potential bias. It is extremely important for 
the individual reading the questiors to avoid using excessive 
voice inflection or expository wording. Time constraints within 
the interview format are a concern, especially in reconciling 
them with the individual subject's needs for patience and a time 
frame that allows for realistic open ended responses. Without a 
doubt, these factors were part of the initial decision to use 
oral questioning only when dictated by the situation. The time 
element and interviewer influence have often been cited as being 
counterproductive {Henerson, Morris and Fitz-Gibbon, 1987) . 

Yet another problem that transcends all types of 
instrumentation is that of confidentiality- Responses that are 
given anonymously increase the chances of getting data that 
generally represents the affective domain of the respondent 
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(Henerson, Morris and Fitz-Gibbon, 1987). While subjects for any 
survey, regardless of physical capacities, are prone to value 
confidence, this is accentuated with the challenged population. 
Evaluators need to remember that this population has often 
experienced a very insensitive side of society. Any instrument 
failing to address confidentiality runs the risk of receiving 
limited or low-quality yields of information, thus impeding tho 
utility of the entire project. 
ISSUES. 

To describe a recipient of special services as a person with 
a disability is not the same as describing him/her as a disabled 
person, h "disability" is limited to certain activities; 
disabled as an adjective implies general inability (Home, 1988) . 
A blind person may have a disability in painting a landscape, but 
with a Braille keyboard may be a better than average computer 
operator. To describe this person as disabled or handicapped is 
doing the person an injustice (Horne, 1988) . If the person is 
not even limitedly disabled by the deficient physical condition 
described as a disability, can it still be called a disability? 
Conceivably, with today's technology and awareness of certain 
disabilities the answer may be "no". Should we call such a 
person "physically challenged" or "exceptional" as suggested by 
Horne (1988)? Maybe not if the physical challenge has been met 
and conguered. in response to an open ended question on our 
survey, respondents did not object to the term "disability" even 
when used in the general sense. However, an interviewee objected 
to being described as having a disability. This subject might 
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still object to the more eropathetic terras »»physically challenge^d" 
or "exceptional" if sincerely self sufficient and truly net 
needing extra services. 

Another important issue in our evaluation was the 
utilization of the evaluation's findings. Because the sponsor 
was following a grant proposal written before any original 
contact with the evaluators, the evaluation had to be described 
as summative. Our question was: how suimnative would an 
evaluation of the first year of a five year program actually be? 
It was rather clear from the start that anything learned from the 
evaluation of the first year would b-^ implemented into the ever- 
developing program. 

For a sponsor who needs an evaluation to satisfy the 
conditions of a grant proposal, summative merit woul-' be the 
appropriate choice in most cases. Summative merit as defined by 
Cuba and Lincoln "is concerned with assessing the intrinsic value 
of some evaluand with the intent of determining whether it meets 
some minimal... standard ...its internal design specifications" 
(Guba and Lincoln 1989) . The grant proposal promises to follow 
some standards or guidelines and the evaluation of the program 
answers how closely it does. It is natural for a grant proposal 
to describe the evaluation of the proposed project in this way; 
however once the project is under way it would be folly to allow 
inadequacies in the program plan to go unaltered on their way to 
fruition if these inadequacies are pointed out through the 
evaluation. 
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There are reasons, such as those related to reliability, why 
an ©valuator may want to save data until the end of the 
evaluation. However, there are considerations such as the rights 
of human subjects as outlined in the Standards of the Joint 
Committee. Denying treatment when it is clearly needed by the 
study's subjects could be considered a violation. If a large 
group (such as the program recipients of a full year) is refused 
access to the improvements of the program, professional ethics 
would insist on the evaluator suggesting the formative 
improvement before the end of the evaluation. This consideration 
alone would not change the focus of the evaluation from suramative 
to formative {Cuba and Lincoln) . However, a new slant is given 
when the sponsor requests immediate feedback from every 
evaluation activity so data can be used at once to improve the 
program. 

Any evaluator could encounter any one of these situations. 
Many probably know their summative evaluation of one year is 
going to be used as justification for formative changes in the 
next year. Often there are program design flaws that need to be 
pointed out and changed right away. Surely, we are not the only 
evaluators whose sponsor has called for summative evaluation, but 
needed formative. It was the encountering of all three 
situations at once that prompts us to call for new terminology to 
explain summative evaluation used for formative purposes. This 
is a support for stake's opinion that the distinction between 
formative and summative is "trivial" (Worthen and Sanders, 1988) . 
The relation of this dilemma to our instrumentation ir an 
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operational as well as an ethical one. Evaluators must carefully 
consider the potential future role of formative data, up front, 
when developing instrumentation. In special needs scenarios, it 
will be a factor even when not overtly articulated, 
STRATEGIES . 

If a survey instrument intended for confidential response 
has to be administered orally, certain procedures could miniTOxze 
the loss of confidentiality. During the first year of the 
evaluation project, it wes discovered that, given a choice, most 
physically challenged subjects would prefer a more anonymous, 
telephonic iriterview. That principle, we suspect, may be 
generalizable to ail people in an overall sense, in 
administering a telephonic interview, the interviewer is 
challenged to hold the interest of the respondent and to make 
smooth transitions between questions. It must be kept in mind 
that telephone conver-ja-Lions can also be laborious for the 
physically challenged subject and that the subject would not 
really be anonymous (since he/she coulu easily be identified), 
However, the perception of anonymity on the part of the s'.:bject 
would probably decrease anxiety and increase the spontaneity of 
responses. By allowing this less stressful option and by 
minimizing evidence of the respondent's identity, the evaluator 
has made a reasonable, professional effort to assure 
confidentiality, as outlined in the Standards of the Joint 
Committee (1981). One final not* about thic strategy is that it 
is only usable if it meets the needs for data input as duiiiieated 
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by the sponsor-ova luntor negotiations. Hence, it has not always 
been feasible in our case. 

When the situation does not allow alternative settings for 
administration, strategies would need to irvolve a sensitivity to 
the subjects' possible anxiety. The findings of Douglas Fuchs on 
the importance of familiarity of test taker with test 
administrator can be generalized to the administration of survey 
instrurtents. When the administration of the test diminishes 
confidentiality, the reliability of the data collected can be 
enhanced if the administrator takes time to develop rapport with 
the subject. Another strategy to glean reliable data from 
surveys of affective responses administered orally is to assure 
the subject repeatedly that he/she does not need to answer any 
question perceived to be inappropriate or intrusive (Standards, 
1981) . 

In answering the problem posed by describing collectively 
the needs of the clients of the program, a strategy to diminish 
the sensitivity of certain terms needs to be developed. A rcviov.' 
of the literature in the areas of exceptionality and disabilities 
would yield some suggestions of empathetic terms and the 
rationale for accepting or rejecting certain terms. Particular 
note should be repeated that a "person with a disability" (even 
from only a semantical point of view) is not a "disabled person" 
(Horne, 1988) This is such a sensitive point that in some cases 
"disability" is rejected as a term in order to not imply the raorc 
comprehensive term "disabled" (Horne, 1988). Our strategy to 
address this problem was to include in our questionnaire an open 
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ended question allowing the subjects to rename the program. We 
presumed the subjects would eliminate any objectionable 
terminology. As noted earlier, clarity and length of items (and 
overall instruments) can be problematic. This boils down to 
evaluators' advance awareness of these items so that proper 
planning will eliminate later anxiety and stimulate more 
applicable results. 
SUMMARY 

In our study and experience we did not develop any 
"cookbook" strategies for dealing with all the identified 
problems. Nor was the discussion of the issues exhaustive. \^e. 
hope though that professional discussion can lead to thorough 
analysis of the unique concerns (problems, issues and strategies) 
involved in evaluating special needs programs. 

Sensitivity to the problems of the sponsor, sensitivity to 
the situation of the program's recipients, sensitivity to the 
cultural pride of the special needs population are a significant- 
part of the evaluation of the special need program. This 
sensitivity almost needs to be innate or intuitive, but with 
effort can be developed by those who want to. There is much 
literature available to help such training in sensitivity. 

Special needs programs have a language, agendas, teaching 
methods that mainstream programs do not. The situations of the 

special needs program can be (and often are) perceived as 
contrary to usual educational prototypes. It is not stretching 
the point too far if we apply this understanding to the special 
needs program evaluation. In this paper we have touched upon 
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some of the areas where the aims and standards of the evaluator 
may be challenged. Our purpose has been to urge caution to the 
evaluator when treading this thin ice. 

Some of the issues we have raised may be settled (or not 
even experienced) by the evaluator who has done his/her homework, 
the prior ethnology. As mentioned before by researching the 
field of educational special needs an evaluator may sensitize 
oneself to the situation of the program being evaluated. 
Furthermore, this sensitivity and familiarity with special needs 
program should lead the evaluator to recognize right away the 
strengths and needs of the program being evaluated by resemblance 
to established models. 

In conclusion, two points we probably all learned in 
kindergarten would help us through the issues and problems 
presented here. The first is: if you don't know, ask. Through 
the established networks of the evaluation community there should 
be someone with the solution to any encountered problem. If not 
then that problem could be the basis for more research. 

The second point is: if it works assume nothing is wrong. 
Do the evaluation the best you can, reporting to the sponsor any 
cause for concern. If the sponsor can use the evaluation's 
results and conclasiors then it can be assumed to be a valid 
study. No one can argue with success. 
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